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System Engineering Research 
Consortium Focus 



□ System Engineering of Complex Systems is not well understood 

□ System Engineering of Complex Systems is Challenging 

System Engineering can produce elegant solutions in some instances 
System Engineering can produce embarrassing failures in some instances 

Within NASA, System Engineering is frequently unable to maintain complex system designs within 
budget, schedule, and performance constraints 

□ “How do we Fix System Engineering?” 

Michael D. Griffin, 61st International Astronautical Congress, Prague, Czech Republic, September 27- 
October 1 , 2010 

Successful practice in System Engineering is frequently based on the ability of the lead system engineer, 
rather than on the approach of system engineering in general 

The rules and properties that govern complex systems are not well defined in order to define system 
elegance 

□ 4 characteristics of system elegance proposed as: 

System Effectiveness 
System Efficiency 
System Robustness 

Minimizing Unintended Consequences 



Approach 



□ Define basis of System Engineering as an Engineering Discipline 


□ Three (3) thrusts to accomplish this 
System Works 

■ Understanding objectively what works in current system development (SLS) 

■ Follow SLS through DCR to study full development phase (once in 40 year opportunity) 

■ Distill the laws governing system engineering in complex systems by studying this approach through 
a combination of academic research with practicing system engineers 

■ Document the laws governing complex system interactions identified in the research 

■ Capture practical guidance from the research in a System Engineering Practitioner’s Guide 

System Design 

■ Apply the laws of governing Complex Systems in an elegant manner in the next major complex 
system development (Mars Transportation System) 

■ Apply and refine guidance in System Engineering Practitioner’s Guide 

System Academy 

■ Train new and practicing system engineers in the engineering basis of the discipline 

■ Preliminary Academic Curriculum 

■ Current work force training courses 



Consortium 



□ Research Process 

Multi-disciplinary research group that spans systems 
engineering areas 

Selected researchers who are product rather than 
process focused 


Missouri University of Science and Technology - System 
Exergy - Dave Riggins 

GWU Space Policy Institute — Application of Policy 
and Law — Zoe Szajnfarber 

Schafer Corporation — Mike Griffin 


□ List of Consortium Members 

UAH — Phil Farrington, Paul Collopy, Dawn Utley, Laird 
Burns, Wes Colley, Bryan Mesmer, Bob Ryan, & Joey 
Shelton 

■ System Cost Modeling - Paul Collopy 

■ System Exergy — PJ Benefield, Bryan Mesmer, Joey 
Shelton 

■ System Complexity, Chief Engineer Interviews - Laird 
Burns 

■ Program/Engineering Decision Making - Dawn Utley 

UCCS- Disciplinary Formalization of System 
Engineering - Stephen Johnson 

Texas A&M — Designing Robust Engineered Systems - 
Richard Malak 

MIT — Informal representation and team decision- 
making in Complex Engineering Systems - Maria Yang 

Iowa State — SE Processes Evaluation - Paul 
Componation 

University of Dayton — System Exergy — John Doty 


□ Previous Consortium Members 

Stevens Institute of Technology — Dinesh Verma 

Spaceworks - John Olds (Cost Modeling Statistics) 

Alabama A&M — Emeka Dunu (Supply Chain 
Management) 

George Mason - John Gero (Agent Based Modeling) 

Oregon State — Irem Turner (Electrical Power Grid 
Robustness) 

Arkansas — David Jensen (Failure Categorization) 


20 graduate students and 3 undergraduate students supported to date 



System Engineering Framework 
Mapping 



Research Framework 


Systems Engineering Focus Areas 


Understanding Mission 

Physics Relationships 

Organization Structure & 
Relationships 

Regulatory Requirements 



Performance 

Cost/ Schedule 

Product Risk 

Organization 


Product Attributes 

Chief Engineer 
Interviews (Burns/UAH) 

Interdisciplinary Design 
Model (Jo hnson/UCCS) 

Physics Based Cost 
Model (Collopy/UAH) 

SE Processes Evaluation 
(Componation/lSU) 

SE Processes Evaluation 
(Componation/lSU) 

Policy and Law 
Implications to System 
Engineering 
(GWU/Szajnf arber ) 


Progra m/E ngi neeri ng 
Decision Making 
(Utley/UAH) 

SE Processes Evaluation 
(Componation/lSU) 

System Entropy and 
Information Entropy 
(Doty/UD, 
Benefield/UAH, 
Colley/UAH) 

SE Processes Evaluation 
(Componation/lSU) 

SE Processes Evaluation 
(Componation/lSU) 

Chief Engineer 
Interviews (Burns/UAH) 


Prog ram /Engineering 
Decision Making 
(Utley/UAH) 

Informal representation 
and team decision- 
making in complex 
engineering systems 
(Yang/MIT) 

Chief Engineer Interviews 
(Burns/UAH) 


System Effectiveness 

Design Robust 
Engineered Systems 
(Malak/Texas A&M) 

Interdisciplinary Design 
Model with Robustness 
Measure 

(Jo hnson/UCCS, Malak 
Texas A&M) 

Physics Based Cost 
Model (Collopy/UAH) 


Physics Based Cost Model 
(Collopy/UAH) 


Robustness 


System Entropy and 
Information Entropy 
(Doty/UD, 

Benefield /UAH, 
Colley/UAH) 

Physics Based Cost 
Model (Collopy/UAH) 


Physics Based Cost Model 
(Collopy/UAH) 

Policy and Law 
Implications to System 
Engineering 
(GWU/Szajnf arber ) 

Efficiency 

Social Model 
(MSFC/Watson) 

Failure Event 
Classification 
(MSFC/Watson, UCCS 
Johnson) 

Physics Based Cost Model 
(Collopy/UAH) 

Social Model 
(MSFC/Watson) 

Failure Event 
Classification 
(MSFC/Watson, UCCS 
Johnson) 

Social Model 
(MSFC/Watson) 


Unintended 

Consequence 


*Chief Engineering Interviews (Bums/UAH) - Applies to all cells in matrix Bold - Primary Research Area 


- Primary Relationship (SE Focus Area x Product Attributes) 


- Secondary Relationship (SE Focus Area x Product Attributes) 


] - Research Gaps at end of Exploration Phase 


System Complexity: Natural or 
Enaineered? 


□ 



Lii'L 


Space Launch 
System (SLS) 




Complex systems are the result of human engineering efforts 
Space Launch System 
Electrical Power Plants 
□ Ships 

Complex systems/Complex Adaptive Systems naturally 
occur in nature 


Cell Structure 


■alisws.<eli 


r,pi*irc£ne 


Elmhurst College 


Cell Structure 
Ecosystems 
Meteorology 
Cosmology 


□ Do these Complex Systems follow the same relationship 
rules? 
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System Complexity: Natural or 
Enaineered or Both? 


□ Do these Complex Systems follow the same relationship rules? 

This is a question still to be answered. 

□ Complex Systems/Complex Adaptive Systems originate from human 
engineering and naturally 

Systems Engineers seek to produce a system for some benefit to society 
and culture (exploration, trade, defense) 

■ These are becoming more complex systems and complex adaptive systems 

■ Complex systems/complex adaptive systems are engineered by natural 
complex systems (organizations) 

Natural systems provide a learning ground for understanding complex 
systems and complex adaptive systems 

■ These natural systems are believed to have the same basic properties of 


complexity that human engineered systems seek to attain 
■ This is a point that needs further definition and mathematical proof 







System Complexity Definition 


□ Complexity: A measure of a system’s intricacy and 
comprehensibleness in interactions within itself and 
with its environment. 



Cell Slruclure 

plasm 

plasms-tei 






System Complexity Properties 


□ Properties: 

Complex systems have a propensity to exhibit unexpected performance 
of intended function 

Complex systems are aggregations of less complex systems 

Complex systems exhibit properties not present in the individual 
subsystems but present in the integration of subsystems (emergent 
property) 

Complex system interactions form networks within the system and with 
the system environments 

■ Complex system interactions can be understood through control theory 
Complex systems exhibit nonlinear responses to system stimuli 

■ Complex systems are difficult to predict 
Complex systems have local optimums 

■ (Organizational Efficiency Determines ability to achieve local optimum) 
Complex systems can be analyzed using two concepts: 

■ laws (rules of interaction) 

■ states (current state and prior history) 



Systems Engineering Complexity 



□ Complex System Properties: 

In general, it may be possible to optimize some CS for known design 
requirements but not for complex adaptive systems (CAS) (Holland, 201 2). 

Predictions of CS and CAS are challenging (Simon, 1 996) 

System engineering teams are organic systems with biological properties 

Perpetual novelty is a characteristic of most CAS (Holland, 2014) 

Recurring patterns are characteristics of CS (Holland, 2014) 

CS can be analyzed using two concepts: laws (rules of interaction) and states 
(current state and prior history) (Holland, 2014) 


*Compiled by UAH/Laird Burns 




System Engineering Complexity 
Modeling Properties 

□ All Engineered Systems are designed, built, and operated to achieve goals. 

Goals are expressions of intent of the designers, builder, and operators. 

The intentions, and hence the goals of system designers, builders, and operators can and 
often do differ. They may or may not be compatible, and may or may not be explicitly 
documented. 

□ Expectations of system behavior are based on models. Models can be formal or 
informal. 

□ Formal modeling of system behaviors using mathematics, physical, and logic is less 
mistake-prone than methods that do not use formal models. This is primarily because 
formal models either reject invalid inputs, or if invalid inputs are accepted, can be 
searched for mistakes, or yield outputs that can be assessed for validity. 

□ Functions are defined as mappings of input states of state variables to output states 
of state variables, y = f(x). 

If the input states map to identical output states of a hypothesized function, there is no 
function. 

Goals are expressed as constraints on the ranges of output state variables of a function. 

Goal = rl < y < rh , where y = f(x). 

Failure is defined as the unacceptable control of an output state variable of a function. 

Failure = rl > y, or y > rh, where y = f(x). 

□ Requirements are formal written statements of goals. 

^Developed by UCCS/Stephen Johnson 



System Engineering Complexity 
Modeling Properties 


□ System goals can be represented hierarchically, due to the hierarchical nature of intentions. 
Some goals define the ultimate intentions for the system, and other goals exist only to support 
them. 

Since goals can be modeled hierarchically, and since goals are expressed as constraints on the ranges 
of output state variables of a function, functions can be modeled hierarchically as well. 

□ A system design implements the means by which functions operate and goals are achieved. 

System design models are not hierarchical. 

State variables are the primary connection between functional and design representations of the 
system, as they exist in both representation types. 

□ Systems theory concepts of hierarchy and recursion apply to DSE, based on the hierarchical 
nature of intentionality. 

□ Control theory is an integral theory of DSE, since goals are achieved by controlling the output 
states of state variables of functions within intended ranges. 

Control of state variables can be achieved either passively through design margins, or actively through 
closed-loop or open-loop control systems. 

Knowledge about the parts of the system is generally constructed and maintained in 
disciplinary and institutional organization structures. 

□ The vast majority of system failures are caused by individual human cognitive or performative 
mistakes, or by human social lack of communication or miscommunication. 

Mistakes due to lack of communication and miscommunication most frequently occur at institutional and 
disciplinary boundaries. 


^Developed by UCCS/Stephen Johnson 


Mars Mission simplified GFT Example 


X, V r Odot, A, Struc T. Cabin 
T. 02 


Functioning 
Crew Performs 
Mars Mission & 
Returns to Earth 


LV "Ready", Struc T, Cabin T r 02 


X, V r 0dot r A, Struc T r Cabin T ? 02 

X, V r 0dot, A r Struc T r Cabin T, 02 
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Des X 
Des V 


Meas X 
Meas V 



0 Err 
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Control 


Measure 

actual 

PosA/el 


Crewed BEO Mission Goal Types 
Transportation 
Crew health and safety 
Scientific and Technical 
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Model Selection Lessons Learned 


□ Biological, Ecological, Meteorological, Zoological 
studies have dealt with large populations/data and 
changes to these populations/data over many 
years 

□ This can generate very large data sets 

□ The statistical analysis techniques for these large 
data sets (big data) provide good insight into 
model selection for complex systems and complex 


Bird Migration Radar, 
Wikipedia 

M ' 



Bar-tailed Godwjt 
Migration, Wikipedia 







Model Selection Lessons Learned 



□ Models must have a scientific basis for their selection 

□ Parsimony 

“Everything should be made as simple as possible, but no simpler” — attributed to Albert Einstein 
Avoid over fitting the data and under fitting the data 

□ Avoid Over Analysis of data sets 

A-priori model candidates defined before data set testing 

Post-priori identification of models can lead to over emphasis of secondary or tertiary effects or the 
under emphasis of primary effects (or the failure to identify these) 

■ “It is widely acknowledged by empirical researchers that data [dredging] is a dangerous practice to be 
avoided...” (White, G. C., “Population viability analysis: data requirements and essential analysis”, in 
Research Techniques in Animal Ecology: Controversies and Consequences”, Boitani, L., Fuller, T. K. (eds.), 
Columbia University Press, 2000) 

■ Over fitting, spurious parameter estimates, inclusion of unimportant variables 

□ Model Selection Bias 

Bias to overemphasize or underemphasize model parameters 

■ Meteorological examples (Miller, A. J, Subset Selection in Regression, Chapman and Hall, London, 1 990) 

□ Model Selection Uncertainty 

Uncertainty in parameter and variable estimates can lead to higher confidence intervals than data 
actually supports 

Reference: Burnham, K. P., Anderson, D. R., Model Selection and Multimodel Inference: A 
Practical Information Theoretic Approach, 2 nd edition. Springer, NY, 2002. 




Concept of Model Complexity 
Supporting Decisions 


Which sub/system What information model is important? 

AICc{F) = -2 (/ Ki (F|C)) + 2K + 


Sensor Inputs (Information) 

Let Sensor A be continuous data (e.g. Temperature) 

Let Sensor B be discrete data (e.g. Fluid Level) 

Let Sensor C be binary data (e.g. Pump ‘on’ = 1, ‘off’ = 0) 


Information Model 




Decision 


Safe? 


Standard Method 


Need ALL sensor data and complex model 
to assess state of system and make 
decision: 

Model = f(A, B, C, AB, AC, BC, ABC) 


Required Sensors: A, B, C 


Exergy-Based Information for Systems 
Analysis, Doty 


Sensors / Location 



AFTER STAGE 4 eyRMQuir 


B -Mt CG AFTER 
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UCG AFTER STAGE i BURNQUT 


C — WHICLE CG 




ri 2 . 3-.-&1 Locablcn i, -3T1, Srfut p 


Proposed Method 


Only include relevant 
information to decision 
with same efficacy: 
Model = f (A) 


Required Sensor: A 
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Summary 


□ Natural Systems provide the best examples of Complex 
Systems and Complex Adaptive Systems 

□ Biological, Ecological, Meteorological, and Zoological 
studies have dealt with large populations and “big 
data” for decades 










